{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MEfAinRUdxmG"
   },
   "source": [
    "# 🚀 Olive Quickstart\n",
    "\n",
    "This notebook shows you how to get started with Olive - an AI model optimization toolkit for the ONNX Runtime. In this notebook, you will:\n",
    "\n",
    "1. Use Olive's automatic model optimizer via a CLI command to optimize an SLM model for the ONNX Runtime (for CPU devices).\n",
    "1. Use the ONNX Runtime Python binding to execute a simple chat interface that consumes the optimized model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PUV-Rwz83L9E"
   },
   "source": [
    "## 🐍 Install Python dependencies\n",
    "\n",
    "First, install the Olive CLI using `pip`:\n",
    "\n",
    "We recommend installing Olive in a [virtual environment](https://docs.python.org/3/library/venv.html) or a [conda environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BmPoU34Vdh6v"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "%pip install olive-ai[auto-opt]\n",
    "%pip install transformers onnxruntime-genai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J7D8rEOx7jnv"
   },
   "source": [
    "## 🪄 Automatic model optimization with Olive\n",
    "\n",
    "In this quickstart you'll be optimizing [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), which has many model files in the Hugging Face repo for different precisions that are not required by Olive.\n",
    "\n",
    "You'll execute Olive's automatic optimizer using the `optimize` CLI command, which will:\n",
    "\n",
    "1. Acquire the model from the the Hugging Face model repo.\n",
    "1. Quantize the model to `int4` using GPTQ.\n",
    "1. Capture the ONNX Graph and store the weights in an ONNX data file.\n",
    "1. Optimize the ONNX Graph.\n",
    "\n",
    "It takes around 60secs to optimize the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lrAGXFeSd_O5"
   },
   "outputs": [],
   "source": [
    "!olive optimize \\\n",
    "    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \\\n",
    "    --precision int4 \\\n",
    "    --output_path models/qwen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xTEL8Zwe7xVn"
   },
   "source": [
    "With the `optimize` command, you can change the input model to one that is available on Hugging Face - for example, to [Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/tree/main) - or a model that resides on local disk. Olive, will go through the same process of *automatically* converting (to ONNX), optimizing the graph and quantizing the weights. The model can be optimized for different providers and devices - for example, you can choose DirectML (for Windows) as the provider and target either the NPU, GPU, or CPU device.\n",
    "\n",
    "If you are using a Hugging Face gated model like Llama-3.2-1B-Instruct then you'll first need to login to Hugging Face using\n",
    "\n",
    "```bash\n",
    "huggingface-cli login --token USER_ACCESS_TOKEN\n",
    "```\n",
    "\n",
    "For more information on user access tokens, [read the Hugging face documentation on user access tokens](https://huggingface.co/docs/hub/security-tokens).\n",
    "\n",
    "## 🧠 Inference model using ONNX Runtime\n",
    "\n",
    "The ONNX Runtime (ORT) is a fast and light-weight package (available in many programming languages) that runs cross-platform. ORT enables you to infuse your AI models into your applications so that inference is handled *on-device*. The following code creates a simple console-based chat interface that inferences your optimized model.\n",
    "\n",
    "### How to use\n",
    "\n",
    "The sample chat app to run is found as [model-chat.py](https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/model-chat.py) in the [onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai/) Github repository."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
